Teleport 18 test plan #55231
Labels
test-plan
A list of tasks required to ship a successful product release.
Comments
Teleport Plugins
Teleport Operator @creack
AWS Node Joining @strideynet
Kubernetes Node Joining @creack
Azure Node Joining @hugoShaka
GCP Node Joining @marcoandredinis
Cloud Labels @marcoandredinis
Passwordless @JoergerThis feature has additional build requirements, so it should be tested with a This sections complements "Users -> Managing MFA devices".
Device Trust @flyinghermitDevice Trust requires Teleport Enterprise. This feature has additional build requirements, so it should be tested with a Client-side enrollment requires a signed Additionally, Device Trust Web requires Teleport Connect to be installed (device A simple formula for testing device authorization is: # Before enrollment.
# Replace with other kinds of access, as appropriate (db, kube, etc)
tsh ssh node-that-requires-device-trust
> ERROR: ssh: rejected: administratively prohibited (unauthorized device)
# Register/enroll the device.
tsh device enroll --current-device
tsh logout; tsh login
# After enrollment
tsh ssh node-that-requires-device-trust
> $
Hardware Key Support @eriktateHardware Key Support is an Enterprise feature and is not available for OSS. You will need a YubiKey 4.3+ to test this feature. This feature has additional build requirements, so it should be tested with a pre-release build (eg: Run all tests on Linux, MacOS, and Windows.
|
Performance @fspmarshall @espadoliniScaling TestScale up the number of nodes/clusters a few times for each configuration below.
Run the ansible-like load test on DynamoDB and CRDB:
Run simulated 30k load tests on all other backends:
Soak TestRun 30 minute soak test directly against direct and tunnel nodes tsh bench ssh --duration=30m user@direct-dial-node ls
tsh bench ssh --duration=30m user@reverse-tunnel-node ls
tsh bench ssh --duration=30m user@foo=bar ls
tsh bench ssh --duration=30m --random user@foo lsConcurrent Session Test
Run a concurrent session test that will spawn 5 interactive sessions per node in the cluster: tsh bench web sessions --max=5000 user ls
tsh bench web sessions --max=5000 --web user ls
Robustness
Teleport with Cloud ProvidersAWS @marcoandredinis
GCP @marcoandredinis
IBM @hugoShaka
Application Access @marcoandredinis
Database Access @greedy52 @gabrielcorado @GavinFrazar @TenerSome tests are marked with "coverved by E2E test" and automatically completed IMPORTANT: please verify health check for every database being tested.
Git Proxy @bernardjkim
TLS Routing @eriktate
Desktop Access @probakowski
|
Binaries / OS compatibilityVerify that our software runs on the minimum supported OS versions as per Windows @fheinecke
Azure offers virtual machines with the Windows 10 2016 LTSB image. This image runs on Windows 10 macOS @doggydogworld
Linux @doggydogworld
Machine ID
With an SSH node registered to the Teleport cluster: @nicholasmarais1158
With a Postgres DB registered to the Teleport cluster: @boxofrad
With a Kubernetes cluster registered to the Teleport cluster: @boxofrad
With a HTTP application registered to the Teleport cluster: @nicholasmarais1158
Host users creation @atburkeHost users creation docs Host users are considered "managed" when they belong to one of the teleport system groups:
CA rotations @capnspacehook
Proxy Peering
SSH Connection Resumption @eriktateVerify that SSH works, and that resumable SSH is not interrupted across a Teleport Cloud tenant upgrade.
Verify that SSH works, and that resumable SSH is not interrupted across a control plane restart (of either the root or the leaf cluster).
EC2 Discovery @marcoandredinis
Azure Discovery @marcoandredinis
GCP Discovery @marcoandredinis
IP Pinning @creackAdd a role with
IGS @smallinsky
Teleport AWS Identity Center Integration @smallinsky @flyinghermit
Teleport SAML Identity Provider @kiosionVerify SAML IdP service provider resource management. Docs: @kiosion
Manage Service Provider (SP) @kiosion
Login and RBAC @kiosion
SAML service provider catalog @kopiczko
SSO MFA @flyinghermitVerify SSO MFA core functionality. The tests below should be performed once Configure both an OIDC connector and a SAML connector following the [Quick GitHub/SAML/OIDC Setup Tips] For simplicity, you can use the same IdP App (client id/secret or entity descriptor) Ensure SSO is allowed as a second factor. The following should work with SSO MFA, automatically opening the SSO MFA redirect URL:
SELinux @JoergerInstall the official SELinux module for Teleport SSH service using
|
GODEBUG='inittrace=1' ./releases/18.0.0-alpha.1/teleport version 2>&1 | rg '^init' | awk '{print $5 " ms " $2}' | sort -n -r | head -10
3.0 ms cloud.google.com/go/compute/apiv1/computepb
1.9 ms github.com/aws/aws-sdk-go/aws/endpoints
1.9 ms github.com/asaskevich/govalidator
1.8 ms github.com/gravitational/teleport/lib/kube/proxy
1.2 ms sigs.k8s.io/structured-merge-diff/v4/typed
1.2 ms github.com/pingcap/tidb/pkg/parser/mysql
1.2 ms cloud.google.com/go/firestore/apiv1/firestorepb
1.1 ms sigs.k8s.io/controller-runtime/pkg/client/apiutil
1.1 ms k8s.io/client-go/kubernetes/scheme
0.76 ms k8s.io/kubectl/pkg/scheme
GODEBUG='inittrace=1' ./releases/18.0.0-alpha.1/teleport version 2>&1 | rg '^init' | awk '{print $8 " bytes " $2}' | sort -n -r | head -10
2235160 bytes github.com/aws/aws-sdk-go/aws/endpoints
1937456 bytes cloud.google.com/go/compute/apiv1/computepb
952328 bytes github.com/asaskevich/govalidator
670360 bytes k8s.io/client-go/kubernetes/scheme
599688 bytes github.com/gravitational/teleport/lib/kube/proxy
545784 bytes k8s.io/kubectl/pkg/scheme
532360 bytes go.opencensus.io/trace/tracestate
529544 bytes go.etcd.io/etcd/api/v3/etcdserverpb
528728 bytes sigs.k8s.io/controller-runtime/pkg/client/apiutil
489600 bytes cloud.google.com/go/firestore/apiv1/firestorepb |
|
I ran into #19037 today when testing GitHub integration. Had to create another GH app. This has been reported quite a while ago, though, so I don't think it's a release blocker. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Manual Testing Plan
Below are the items that should be manually tested with each release of Teleport.
These tests should be run on both a fresh installation of the version to be released
as well as an upgrade of the previous version of Teleport.
Adding nodes to a cluster @danielashare
Labels @danielashare
server_infoTrusted Clusters @eriktate
RBAC @bernardjkim
Make sure that invalid and valid attempts are reflected in audit log. Do this with both Teleport and Agentless nodes.
Verify that custom PAM environment variables are available as expected. @capnspacehook
Users @bl-nero
With every user combination, try to login and signup with invalid second
factor, invalid password to see how the system reacts.
WebAuthn in the release
tshbinary is implemented using libfido2 forlinux/macOS. Ask for a statically built pre-release binary for realistic
tests. (
tsh fido2 diagshould work in our binary.) Webauthn in Windowsbuild is implemented using
webauthn.dll. (tsh webauthn diagwithsecurity key selected in dialog should work.)
Touch ID requires a signed
tsh, ask for a signed pre-release binary so youmay run the tests.
Windows Webauthn requires Windows 10 19H1 and device capable of Windows
Hello.
Adding Users OTP
Adding Users WebAuthn
Adding Users via platform authenticator
Managing MFA devices
tsh mfa addtsh mfa addtsh mfa addtsh mfa lstsh mfa rmtsh mfa rmLogin with MFA
tsh mfa addLogin OIDC
Login SAML
Login GitHub
Deleting Users
Backends @hugoShaka
Session Recording @Joerger
Enhanced Session Recording @Joerger
disk,commandandnetworkevents are being logged.enhanced_recordingrole option.Auditd @Joerger
teleport/lib/auditd/common.go
Lines 25 to 34 in 7744f72
Audit Log @eriktate
Audit log with dynamodb
Audit log with Firestore
Failed login attempts are recorded
Interactive sessions have the correct Server ID
server_idis the ID of the node in "session_recording: node" modeserver_idis the ID of the node in "session_recording: proxy" modeforwarded_byis the ID of the proxy in "session_recording: proxy" modeNode/Proxy ID may be found at
/var/lib/teleport/host_uuidin thecorresponding machine.
Node IDs may also be queried via
tctl nodes ls.Exec commands are recorded
scpcommands are recordedSubsystem results are recorded
Subsystem testing may be achieved using both
Recording Proxy mode
and
OpenSSH integration.
Assuming the proxy is
proxy.example.com:3023andnode1is a node runningOpenSSH/sshd, you may use the following command to trigger a subsystem audit
log:
sftp -o "ProxyCommand ssh -o 'ForwardAgent yes' -p 3023 %r@proxy.example.com -s proxy:%h:%p" root@node1External Audit Storage @marcoandredinis
External Audit Storage must be tested on an Enterprise Cloud tenant.
Instructions for deploying a custom release to a cloud staging tenant: https://github.com/gravitational/teleport.e/blob/master/dev-deploy.md
tsh play <session-id>worksInteract with a cluster using
tsh@danielashareThese commands should ideally be tested for recording and non-recording modes as they are implemented in a different ways.
Interact with a cluster using
ssh@danielashareMake sure to test both recording and regular proxy modes.
Verify proxy jump functionality @atburke
Log into leaf cluster via root, shut down the root proxy and verify proxy jump works.
Interact with a cluster using the Web UI @atburke
X11 Forwarding @eriktate
xeyesandxclip:apt install x11-apps xclipxeyes. Thenbrew install xclip.ssh_service.x11.enabled = yestsh ssh -X user@node xeyestsh ssh -X root@node xeyestsh ssh -Y server01 "echo Hello World | xclip -sel c && xclip -sel c -o"should print "Hello World"tsh ssh -X server01 "echo Hello World | xclip -sel c && xclip -sel c -o"should fail with "BadAccess" X errorUser accounting @atburke
/var/run/utmpon Linux./var/log/wtmpon Linux.Combinations @timothyb89
For some manual testing, many combinations need to be tested. For example, for
interactive sessions the 12 combinations are below.
Add an agentless Node in a local cluster.
Add a Teleport Node in a local cluster.
Add an agentless Node in a remote (leaf) cluster.
Add a Teleport Node in a remote (leaf) cluster.
Teleport with EKS/GKE @creack
Teleport with multiple Kubernetes clusters @creack
Note: you can use GKE or EKS or minikube to run Kubernetes clusters.
Minikube is the only caveat - it's not reachable publicly so don't run a proxy there.
tsh login, check thattsh kube lshas your clusterkubectl get nodes,kubectl exec -it $SOME_POD -- shtsh login, check thattsh kube lshas your clusterkubectl get nodes,kubectl exec -it $SOME_POD -- shtsh login, check thattsh kube lshas your clusterkubectl get nodes,kubectl exec -it $SOME_POD -- shtsh login, check thattsh kube lshas both clusterstsh kube loginkubectl get nodes,kubectl exec -it $SOME_POD -- shon the new clustertsh login, check thattsh kube lshas all clustersnameandlabelsStep 2login value matching the rowsnamecolumnnameorlabelsin the search bar worksnamecolumKubernetes exec via WebSockets/SPDY @creack
To control usage of websockets on kubectl side environment variable
KUBECTL_REMOTE_COMMAND_WEBSOCKETScan be used:KUBECTL_REMOTE_COMMAND_WEBSOCKETS=true kubectl -v 8 exec -n namespace podName -- /bin/bash --version. With-v 8logging levelyou should be able to see
X-Stream-Protocol-Version: v5.channel.k8s.ioin case kubectl is connected over websockets to Teleport.To do tests you'll need kubectl version at least 1.29, Kubernetes cluster v1.29 or less (doesn't support websockets stream protocol v5)
and cluster v1.30 (does support it by default) and to access them both through kube agent and kubeconfig each.
KUBECTL_REMOTE_COMMAND_WEBSOCKETS=falseKUBECTL_REMOTE_COMMAND_WEBSOCKETS=trueX-Stream-Protocol-Version: v5.channel.k8s.io)X-Stream-Protocol-Version: v5.channel.k8s.io)Kubernetes auto-discovery @creack
tctl create.tctl create -f.tctl rm.Kubernetes Secret Storage @creack
StatefulsetKubernetes RBAC @tigrato
Kubernetes resources
Create some namespaces:
ns.yaml:
Create some deployments:
deployments.yaml:
Create some pods:
pods.yaml
Create some CRDs (namespaced and cluster-wide):
crds.yaml
Create some custom resources:
crs.yaml
Restart the kubernetes_service after creating CRDs.
NOTE: Unless specified otherwise, the
verbfield ofkubernetes_resourceschema needs to be set asverbs: ["*"]pod, but should also be tested withdeploymentandclusterrole:kubernetes_resources:{"kind":"pod","name":"*","namespace":"*"}- must allow access to every pod{"kind":"pod","name":"somename","namespace":"*"}- must allow access to podsomenamein every namespace{"kind":"pod","name":"*","namespace":"<somenamespace>"}- must allow access to any pod in<somenamespace>namespace*wildcards -myapp-*and regex fornameandnamespacefieldskubectl delete --raw=/api/v1/namespaces/<namespace name>/pods)kubernetes_resources:kubernetes_groupsthat denies exec into a pod{"kind":"*","name":"*","namespace":"foo"}:pods,deploymentsin thefoonamespaceclusterroles,nodescrontabsand cluster-wide CRDglobalsnamespacekind{"kind":"namespace","name":"foo"}(different behavior than rolev8)pods,deploymentsin thefoonamespacecrontabsin thefoonamespaceclusterrolesandnodescrontabsin any namespace{"kind":"crontabs","api_group":"stable.example.com","namespace":"foo",...}foonamespace and no acess indevandprodnamespaces.{"kind":"crontabs","api_group":"*.example.com","namespace":"dev",...}devnamespaceglobals{"kind":"globals","namespace":"",...}(missing api_group){"kind":"globals","api_group":"*","namespace":"",...}globalscluster-wide resource{"kind":"namespaces","namespace":"foo",...}{"kind":"*","name":"*","namespace":"*","api_group":"*","verbs":["*"]}pods,deployments, including namespaced CRDcrontabsclusterroles,nodes, including cluster-wide CRDglobals{"kind":"*","name":"*","namespace":"","api_group":"*","verbs":["*"]}clusterroles,nodes, including cluster-wide CRDsglobalspods,deployments,services, including namespaced CRDsglobals{"kind":"*","name":"*","namespace":"*","api_group":"*","verbs":["*"]}{"kind":"crontabs","name":"*","namespace":"*","api_group":"*","verbs":["*"]}{"kind":"crontabs","name":"*","namespace":"","api_group":"stable.example.com","verbs":["*"]}ex:
{"kind":"pods","name":"*","namespace":"*"},{"kind":"clusterroles","name":"*"},{"kind":"deplyments","name":"*","namespace":"*"}.services,nodes,crontabs,globals.ex:
{"kind":"pods","name":"*","namespace":"*"},{"kind":"crontabs","api_group":"*","name":"*","namespace":"*"}pods,crontabsandother resourcesservices,globals,nodes.{"kind":"namespaces","name":"foo","verbs":["*"]}{"kind":"*","api_group":"*","name":"*","namespace":"","verbs":["*"]}search_as_rolesis not allowed.Teleport with FIPS mode @capnspacehook
ACME @capnspacehook
Migrations @hugoShaka
SSH should work for both main and old clusters
SSH should work
Command Templates
When interacting with a cluster, the following command templates are useful:
OpenSSH
Teleport
Teleport with SSO Providers
GitHub External SSO @greedy52
tctl ssofamily of commands @flyinghermitFor help with setting up sso connectors, check out the [Quick GitHub/SAML/OIDC Setup Tips]
tctl sso configurehelps to construct a valid connector definition:tctl sso configure github ...creates valid connector definitions @greedy52tctl sso configure oidc ...creates valid connector definitionstctl sso configure saml ...creates valid connector definitionstctl sso testtest a provided connector definition, which can be loaded fromfile or piped in with
tctl sso configureortctl get --with-secrets. Validconnectors are accepted, invalid are rejected with sensible error messages.
tctl sso test.SSO login on remote host @flyinghermit
tshshould be running on a remote host (e.g. over an SSH session) and use thelocal browser to complete and SSO login. Run
tsh login --callback <remote.host>:<port> --bind-addr localhost:<port> --auth <auth>on the remote host. Note that the
--callbackURL must be able to resolve to the--bind-addrover HTTPS.SAML SSO login with different binding methods @flyinghermit
http-redirect. Verify SAML authentication request is sent in a URL.preferred_request_binding: http-redirectvalue in the SAML connector spec.http-post. Verify SAML authentication request is sent in an HTML form.preferred_request_binding: http-postvalue in the SSO connector spec.http-redirectrequest.The text was updated successfully, but these errors were encountered: